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2.  Research  Program  Objectives 


In  the  design  of  visual  displays  it  is  important  that  different 
types  of  information  be  presented  free  of  interference  with  each  other. 

It  is  well  known  that  one  visual  stimulus  can  mask  the  detection  of 
another  if  they  are  similar  in  spatial  and  temporal  characteristics. 

Such  interference  diminishes  with  increasing  stimulus  difference,  and 
interference  below  some  acceptably  low  level  can  be  used  as  a  design 
criterion. 

Continuous  information  transfer  requires  a  varying,  unpredictable 
stimulus.  To  assess  the  masking  properties  of  an  information-containing 
visual  field  it  is  necessary  that  the  mask  be  in  some  sense  random.  We  have 
interpreted  the  limited  existing  data  on  random  (noise)  masking  as  implying 
a  greater  potency  to  the  finite  bandwidth  random  mask  than  to  the  nonrandom 
(single-component)  grating. 

Both  the  test  and  mask  stimuli  are  characterized  in  a  3-dimensional 
'spatio-temporal  frequency  space.  The  test  stimulus  is  in  all  instances 
a  traveling-wave  sinusoidal  grating  (i.e.,  has  a  single  punctate  repre¬ 
sentation  in  frequency  space,  together  with  the  necessary  conjugate  point). 

The  mask  stimulus  is  either  coherent  (i.e.,  a  single,  traveling  sinusoidal 
wave)  or  incoherent  (i.e.,  occupies  a  finite  3-dimensional  band  volume  in 
frequency  space) .  Since  3  parameters  are  required  to  specify  the  test 
stimulus,  while  9  parameters  are  required  for  the  incoherent  mask  (3 
reference  frequencies  for  the  centroid  of  the  mask  spectral  power  and 
6  second  moments  of  spectral  power  with  respect  to  the  centroid),  a  complete 
study  would  involve  a  total  of  12  parameters.  To  address  the  overwhelming 
complexity  of  a  12  parameter  study  we  resort  to  "cuts"  of  reduced  dimen¬ 
sionality.  For  example,  the  use  of  a  coherent  mask  reduces  the  dimensionality 
from  12  to  6.  However  an  essential  objective  of  the  research  is  the 
characterization  of  the  anticipated  major  increase  of  masking  efficacy 
when  the  mask  is  incoherent  rather  than  coherent.  Therefore  the  results 
of  the  coherent  mask  studies  are  used  to  choose  the  most  relevant 
combination  of  parameters  for  the  incoherent  masking  experiments. 
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3.  Status  of  the  Research 
3.1  Coherent  Masking  Studies 

Underlying  our  original  Proposal  was  the  concept  that  masking 
bands  would  generally  be  symmetrical,  with  respect  to  spatial  and 
temporal  frequencies,  centered  on  the  masking  wave  vector  (Proposal 
Appendix  C) ,  and  that  the  major  interest  would  be  the  bandwid:hs  of 
masking  for  various  "cuts"  through  three-dimensional  spatio-temporal 
frequency  space.  We  began  with  knowledge  of  one  2D  "cut"  in  3D  space 
(namely  for  f^  =  0,  so  all  components  were  stationary),  and  next 
explored  another  "cut"  (namely  0  =  0,  so  all  components  shared  the  same 
orientation) .  Due  to  delays  in  delivery  of  our  own  equipment  we  began 
these  studies  with  borrowed  facilities  (M.I.T.  Manned  Vehicle  Laboratory), 
using  a  simpler  psychophysical  procedure  which  was  a  modified  method  of 
adjustment.  Our  first  spatio-temporal  "cut"  in  3D  space,  using  travelling- 
wave  mask  and  test  components,  revealed  extreme  asymmetry  of  masking, 
the  mask  (4  c/ deg,  8  Hz)  very  strongly  masked  patterns  of  lower  spatial 
and 'temporal  frequency,  while  facilitating  the  detection  of  higher 
spatial  frequency  test  patterns.  (See  1982  Informal  Progress  Report, 

Appendix  A.)  We  also  discovered  that  the  strongest  masking  effects 
occur  when  the  mask  and  test  components  are  matched  in  velocity,  so  that 
they  both  have  the  same  ratio  of  temporal  to  spatial  frequencies. 

In  brief,  we  have  found  that  the  addition  of  sustained,  travelling- 
wave  motion  profoundly  affects  the  entire  masking  phenomenon.  It  is 
important  that  these  results  be  precisely  documented  for  publication. 

We  have  been  able  to  confirm  these  results  using  signal  detection  theory 
with  forced-choice  "staircases",  and  we  are  currently  exploring  in 
further  detail  the  full  extent  of  the  facilitation  regime.  We  anticipate 
conclusion  of  this  particular  phase  of  work  by  the  end  of  this  contract 
year.  The  completion  of  these  coherent  masking  studies  will  entail 
exploring  the  full  3D  masking  function,  in  the  same  travelling-wave 
paradigm,  in  order  to  characterize  the  band-volumes  of  sensitivity  reduction 
and  enhancement  in  spatio-temporal  frequency  space. 

i» 


3.2  Finite  Bandwidth  Noise  Masking 

As  outlined  in  the  Proposal,  we  are  approximating  continuous  band- 
limited  two-dimensional  noise  Images  by  ensmebles  of  discrete  points  in 
ttie  spatial  frequency  domain  (punctate  2D  spectra) ,  due  to  data  rate 
limitations  in  computer  controlled  displays  of  reasonable  cost.  We  have 
developed  a  powerful  and  flexible  system  of  hardware  and  software  which 
synthesizes  dynamic  two-dimensional  visual  patterns  from  such  ensembles  of 
Fourier  components,  defined  in  a  three-dimensional  spatio-temporal  frequency 
space  such  that  each  combination  o f  spatial  frequency,  orientation,  and 
temporal  frequency  has  an  associated  amplitude  and  phase.  At  our  current 
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image  generation  frame  rate  of  250  frames/second,  we  can  synthesize 
spatio-temporal  textures  containing  as  many  as  20  such  Fourier  components 
when  two  components  of  matching  orientation  are  presented  in  each  frame. 
Further  details  concerning  these  developments  may  be  found  in  Appendix  C. 

With  the  computer  and  display  hardware  in  place  our  first  objective 
was  to  determine  how  few  punctate  spectral  components  were  required  to 
simulate  dynamic  bandlimited  2D  noise.  We  conjectured  in  the  Proposal 
that  as  few  as  16  might  suffice,  and  we  were  prepared  to  find  that  per¬ 
haps  as  many  as  50  would  be  required.  We  were  amazed  to  discover  that 
the  true  minimum  could  be  as  few  as  6,  if  they  were  suitably  chosen 
within  relatively  narrow  bandwidths  (of  about  one  octave  of  spatial 
frequency  and  30°  of  orientation) .  As  the  bandwidth  of  the  2D  noise  to 
be  simulated  increases,  the  required  number  of  components  increases,  and 
the  principle  appears  to  be  governed  by  the  psychophysical  2D  masking 
bandwidth  based  on  a  single  Fourier  component  (the  putative  2D  channel 
band  area).  Thus,  as  long  as  the  punctate  spectral  components  lie  within 
the  coherent  masking  bands  of  one  another,  and  as  long  as  the  number  of 
discrete  Fourier  components  is  6  or  more,  their  sum  will  be  approximately 
indistinguishable  from  continuous,  stationary  bandlimited  2D  spatial  noise. 
This  point  is  illustrated  for  both  isotropic  and  anisotropic  spectra  in 
Figure  1.  It  appears  that  even  fewer  components  may  be  required  for 
simulating  dynamic  2D  noise,  since  the  different  temporal  frequencies 
associated  with  each  Fourier  component  remove  coherence  of  the  image  across 
time.  Since  these  effects  can  only  be  appreciated  in  a  dynamic  visual 
display,  we  have  prepared  a  videotape  and  an  accompanying  descriptive 
brochure  which  has  been  submitted  to  the  contract  monitor.  The  final 
section  of  the  videotape  presents  an  isotropic  spatio-temporal  texture 
synthesized  from  the  same  10  components  (but  with  temporal  frequency 
included)  as  used  in  the  isotropic  texture  shown  in  Figure  lb.  The 
dynamic  isotropic  texture  appears  even  more  incoherent  than  the  static 
one,  even  though  both  have  the  same  punctate  spatial  spectrum. 

A  basic  premise  of  our  original  Proposal  was  that  finite  band  noise 
masking  was  more  effective  than  very  narrow  band  or  coherent  masking.  Our 
basic  experiments  have  already  confirmed  this.  For  example,  if  a  Subject 
cannot  distinguish  between  patterns  composed  of  5  punctate  components  vs.  6, 
it  means  that  the  sixth  component  is  effectively  'masked'  by  the  other  5. 
Since  all  components  have  the  same  contrast,  C,  the  5  component  texture 
has  an  overall  (expected  peak-to-peak)  contrast  of  C  /5  which  means  that 
any  single  component  is  undetectable  in  a  mask  whose  contrast  is  only  *5 
larger.  This  is  far  more  effective  than  coherent  masking. 

The  1983  Informal  Progress  Report  (Appendix  B  here)  describes  how 
the  band-limited  stimulus  gives  the  percept  of  modulation  "net"  overlying 
a  carrier  wave,  and  discusses  the  implications  of  this  percept  for 
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FIGURE  1  Example  of 


(a)  a  six-component  anisotropic  noise  pattern,  and 

(b)  a  10-component  isotropic  noise  pattern,  with  punctate  2D 
spectra  as  indicated  in  the  Fourier  plane  (left  Panel  each  Figure 
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identifying  essential  underlying  nonlinearities  in  visual  system 
processing.  When  the  band-limited  stimulus  is  composed  of  travelling- 
wave  components  the  variety  of  percepts  is  remarkable.  For  example,  the 
modulation  net  and  the  carrier  can  have  completely  different  perceived 
directions  of  motion,  leading  to  a  bi-stable  motion  percept.  This  is 
documented  in  the  videotape  mentioned  above  which  was  presented  at  the 
1983  ARVO  meeting  and  in  an  expanded  form  will  be  shown  at  the  6th 
European  Conference  on  Visual  Perception.  This  videotape  together 
with  its  supporting  descriptive  brochure,  constitutes  an  essential 
part  of  this  Progress  Report. 
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Background 


Since  this  is  an  initial  report  covering  only  8  months  of 

activity  it  is  appropriate  to  review  briefly  the  objectives  of  the 

three-year  study.  Simply  stated  we  intend  to  measure  the  detectability 

of  a  sinusoidal  traveling-wave  grating  in  the  presence  of  a  mask  of 

band-limited  spatiotemporal  noise.  The  mask  is  specified  in  terms  of 

three  spectral  variables:  temporal  frequency,  f^,  and  two  spatial 

frequencies,  f  and  f  .  Alternatively,  the  two  spatial  frequencies  can 
x  y 

be  replaced  by  a  single  spatial  frequency,  and  a  wave  orientation,  f 

s 

and  0,  where  f  =  f  cos  0  and  f  =  f  sin  0.  A  total  of  six 
x  s  y  s 

quantities  are  required  to  specify  the  mask:  three  quantities  defining 
the  spectral  center  of  the  noise  band  and  three  bandwidths. 


Some  rather  limited  existing  data  have  suggested  that  a  noise  mask 
which  is  temporally  broad-band  is  considerably  more  effective  at  a 
given  contrast  level  than  is  a  temporally  coherent  mask.  Our  aim  is 
to  study  this  question  in  the  general  context  of  the  full  three  spectral 
dimensions  and  to  ascertain  for  which  bandwidths  the  masking  property 
makes  the  transition  from  the  less  effective  "coherent"  state  to  the 
more  effective  "random"  state. 


The  generation  of  band-limited  noise  masks  in  real  time  is  a 
technically  challenging  one.  The  method  we  are  developing  consists  of 
storing,  in  a  computer,  sequences  of  temporally  correlated  numbers  which 
represent  the  amplitudes  of  discrete  spatial  frequency  components  (i.e. , 
individual  sine  wave  gratings) .  These  sine  wave  gratings  are  generated 
on  a  CRT  by  preprogrammed  electronics  and  a  large  number  can  be  combined 
by  frame  interleaving.  To  avoid  the  contrast  degradation  resulting  from 
many  interleaved  frames  we  are  electronically  summing  gratings  of  matched 
orientation  before  generating  the  CRT  representation. 


Hardware  Development 

The  experimental  apparatus  consists  of  the  controlling  digital 
computer,  the  special  purpose  electronics  for  grating  generation  and  the 
display  unit.  We  have  acquired  a  PDP  11-34  computer  which  is  installed 


and  running  with  an  RT-11  operating  system.  It  happened  that  the  funds 
requested  in  our  research  proposal  (and  allocated)  were  insufficient  to 
purchase  a  new  computer  with  essential  peripherals  at  current  prices. 
Consequently  we  purchased  a  used  computer  and  are  pleased  to  find  that 
it  performs  excellently.  In  order  effect  a  long  range  economy  we 
requested  permission  to  expend  first-year  funds  for  an  in-lab  graphics 
capability  so  that  data  evaluation  and  presentation  could  be  done 
efficiently  and  cheaply.  This  equipment  is  now  in  place. 

We  have  had  a  temporary  bottleneck  in  the  construction  of  the 
specialized  electronics.  Almost  all  of  the  components  are  now  on  hand 
and  we  expect  to  do  prototype  testing  in  6  to  8  weeks.  The  display 
unit,  a  Tektronix  608,  is  in  place. 


Preliminary  Studies 


There  are  two  directions  in  which  we  have  been  able  to  proceed 
before  the  availability  of  the  computer  generated  noise  mask.  The  first 
is  to  extend  the  study  of  coherent  masking  to  three  spectral  dimensions. 
Since  the  coherent  mask  represents  the  limit  of  zero-bandwidth  random 
masking  it  is  a  situation  which  must  necessarily  be  understood  as  a 
reference  for  all  finite  bandwidth  studies.  The  second  direction  is  to 
make  estimates  of  how  many  discrete  components  will  be  required  within 
any  finite  three-dimensional  spectral  band  to  give  the  appearance  of 
randomness.  This  will  serve  as  a  guide  to  the  adequacy  of  our  electronics 
hardware  package. 


Coherent  Masking  Results 


In  our  studies  we  have  elected  to  hold 


f  ,  f  , 
x’  y’ 


and  f  of  the  mask 


fixed  and  vary  f  ,  f  ,  and  f  of  the  test  grating.  A  relatively  simple 
x  y  t 

transformation  can  convert  these  results  to  the  inverse  case  if  desired. 


Rather  than  plunge  into  the  full  3-dimensional  variation  of  the  test 


f  ,  f  and  f  we  have  elected  to  take  2-dimensional  cuts  In  the  3-space, 
x  y  t 

The  first  such  cut  Is  to  make  f  of  the  test  match  that  of  the  mask  and 
vary  only  f^  and  f^.  The  study  of  the  special  sub-case  where  both  the 
mask  and  test  have  ffc  =  0  (i.e.,  the  mask  and  test  are  stationary  and 
the  test  is  turned  on  and  off  slowly)  was  begun  before  this  research 
contract  by  one  of  us  (J.G.  Daugman)  and  has  been  brought  to  completion 
during  this  reporting  period.  The  key  results  are  summarized  in  an  abstract 
prepared  for  the  1982  ARVO  meeting  together  with  a  set  of  data  figures 
(both  attached) .  These  results  show  to  be  fallacious  the  commonly 
preconceived  view  that  such  a  2-dimensional  masking  function  should  be 
polar  separable  (i.e.,  the  product  of  a  function  solely  of  spatial 
frequency  with  a  function  solely  of  orientation).  Moreover  they  show 
that  the  masking  bandwidth  for  gratings  which  are  colinear  is  about  twice  as 
broad  as  for  gratings  which  have  the  same  frequency  but  differ  in 
orientation. 

The  second  2-dimensional  cut  in  3-space  we  have  studied  is  to 

make  the  orientation  of  the  test  match  the  mask  and  vary  only  fg  and 

ffc  (i.e.,  the  mask  and  test  are  colinear  waves  traveling  in  the  same 

direction  but  with  different  f  and  f  ) .  We  have  made  extensive  studies 

s  c 

with  three  subjects.  The  results,  summarized  in  another  abstract  for 
ARVO,  are  attached,  along  with  a  contour  plot  of  the  masking  function. 

When  the  mask  and  test  match  in  ffc  the  masking  is  reasonably  symmetric 
in  f  (as  we  had  anticipated) ,  but  when  they  match  in  f  the  masking  is 
very  asymmetrical  in  ft.  What  is  especially  surprising  is  that  when  the 
mask  and  test  match  in  drift  velocity  there  is  a  very  great  asymmetry 
(actual  facilitation  for  test  frequencies  higher  than  the  mask).  One 
fact  is  clear:  the  subjects  did  not  follow  the  traveling-wave  mask  with 
smooth  eye  movement,  for  had  they  done  so  the  detection  of  the  matched- 
velocity  test  vould  have  given  a  much  more  symmetrical  masking  function. 

We  are  in  the  process  of  thinking  through  the  implications  of  these  unusual 
findings  for  our  projected  random  mask  studies. 


Achieving  Randomness  with  Few  Spectral  Components 


Previous  work  has  suggested  that  four  colinear  gratings  with 
irrational  frequencies  can  give  a  good  impression  of  randomness.  Our 
preliminary  studies  confirm  this  but  go  beyond  it.  We  have  found  that 
for  only  two  components,  with  frequencies  in  the  ratio  of  1.61,  it  is 
difficult  for  the  observer  to  be  specific  about  the  nature  of  the 
intrinsic  regularity.  There  appear  to  be  selected  frequency  ratios, 

1.61  and  others,  which  convey  the  impression  of  maximum  irregularity 
and  we  plan  to  document  this  more  precisely.  However,  two  gratings  with 
different  orientations  always  give  an  identifiable  regular  pattern 
regardless  of  their  spatial  frequencies.  The  reason  is  that  where  two 
orientations  are  concerned  what  the  brain  perceives  is  the  difference 
of  the  two  wavevectors  (i.e.  the  low  spatial  frequency  modulation 
present  in  the  combined  pattern).  Thus,  two  components  reduce  to  one 
net  percept.  We  are  now  testing  whether  three  grating  components, 
so  oriented  that  the  difference  wavevectors  are  related  in  some  optimal 
Irrational  way,  might  give  an  appearance  of  irregularity  analogous  to 
the  case  of  two  colinear  gratings.  For  the  present  all  we  can  say  is 
that  the  appearance  of  randomness  in  orientation  is  less  easy  to  achieve 
than  in  spatial  frequency.  The  implication  for  the  generation  of  band 
limited  noise  masks  is  that  we  may  have  to  use  the  full  complement  of  10 
orientations  originally  planned. 
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POLAR  SPECTRAL  NONSEPARABILITY  OF  TWO-DIMENSIONAL  SPATIAL 
FREQUENCY  CHANNELS.  John  G.  Daugman,  Harvard  University. 

Spatial  frequency  channels  are  usually  conceptualized  as 
the  product  of  a  spatial  frequency  tuning  curve  times  an 
orientation  tuning  curve.  This  implicit  assumption  of  polar 
spectral  separability  was  examined  by  2AFC  masking  experi¬ 
ments  in  which  the  superimposed  mask  and  test  gratings  span¬ 
ned  the  2D  Fourier  plane  (i.e.  all  relative  orientations  and 
spatial  frequencies).  A  2D  channel  tuning  surface  was  thus 
constructed  over  the  Fourier  plane,  interpolated  from  a  Car¬ 
tesian  grid  of  121  sampling  points,  for  each  of  four  subjects. 

These  channel  tuning  surfaces  have  iso-sensitivity  contours 
which  are  roughly  elliptical,  but  they  are  neither  Cartesian 
nor  polar  separable.  The  orientation  bandwidth  of  a  single 
channel  changes  dramatically  depending  on  the  probe  spatial 
frequency  relative  to  the  channel’s  preferred  frequency. 

For  all  four  subjects  there  was  a  monotonically  decreasing 
relationship  between  orientation  bandwidth  and  probe  spatial 
frequency,  ranging  from  an  orientation  half-width  of  32 
degrees  when  probing  1.5  octaves  below  the  channel's  prefer¬ 
red  frequency  to  a  half-width  of  6  degrees  when  probing  one 
octave  above.  Thus  a  spatial  visual  channel  is  manifestly 
not  expressible  as  the  product  of  a  spatial  frequency  tun¬ 
ing  curve  times  an  orientation  tuning  curve. 

Inverse  2D  Fourier  transformation  of  the  data  reveal  the 
underlying  even  and  odd  space-domain  kernels  (point-spread 
functions)  of  the  2D  channels,  which  resemble  classic  elon¬ 
gated  center-surround  receptive  fields.  These  characteris¬ 
tics  of  anisotropic  localized  visual  filters  are  analyzed 
in  terras  of  2D  Gabor  theory  of  joint  uncertainty  minimiza¬ 
tion.  (Supported  by  AFOSR  Contract  if  F49620-81-K-0016 ) . 
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; SPATIOTEMPORAL  MASKING:  ASYMMETRY,  NONSEPARABILITY  AND 
i FACILITATION.  Y.Y.  Zeevi,  R.E.  Kronauer,  J.D.  Daugman. 
i  Division  of  Applied  Sciences,  Harvard  University. 

I 

We  have  studied  masking  between  sinsoidal  gratings, 
drifting  codirectionally ,  which  may  differ  in  both  drift 
frequency,  ft#  and  spatial  frequency,  fs.  The  mask  is 
ever-present.  The  test  grating  pulse  is  250  msec  (co- 
sined  envelope)  preceded  by  a  warning  tone.  Contrast 
threshold,  Cth  is  measured  by  a  modified  staircase  pro¬ 
cedure.  Masking  effectiveness,  m,  is  the  ratio  of  Cth 
with  mask  to  Cth  without  mask. 

i  A  40%  contrast  mask  with  fs  =  4  cpd  and  f .  *  8  hz 
gives,  for  a  test  with  matching  ft»fs  P~esented  in  phase, 
m=20.  For  test  gratings  which  match  the  mask  in  ft  the 
masking  effectiveness  is  symmetric  in  fs  (on  a  linear 
plot)  with  m=10  found  at  about  2  cpd  and  6  cpd.  For  test 
gratings  which  match  the  mask  in  fs%(fs=4cpd)  m  is  very 
•asymmetric;  m  drops  sharply  for  f*>8  hz  (m=10  at  f.=12 
hz)  but  is  sustained  for  f^<8  hz  (m  *  14  at  ft  =  2766 
hz).  For  test  gratings  which  match  the  mask  drift 
velocity  (and  presented  in  cosine  phase  when  harmonically 
related)  the  masking  asymmetry  is  even  stronger.  The 
detection  of  3rd  harmonic  test  gratings  (f_=12cpd, 
ffc=24hz)  is  strongly  facilitated  (m<.3)  while  the 
detection  of  1/3  subharmonic  (f  =1.33  cpd,  ft=2.66  hz) 
is  strongly  masked  (m=14) . 

Thus,  the  spa tio temporal  masking  surface  m(fs,ft)is 
not  separable  into  a  product  of  functions  of  fs  and  ft 
alone . 

(Supported  by  AFOSR  contract  F49620-81-K-0016 . ) 
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INFORMAL  SCIENTIFIC  REPORT 

Contract  No.  F49620-81-K-0016  "Quantification  of  Interference  and 
Detectability  Properties  of  Stimuli  for  Optimal  Display  Design." 

Our  research  is  directed  toward  quantifying  the  masking  effects  of 
stochastic  spatio-temporal  images.  Since  there  are  very  many  parameters 
in  such  a  study  we  began  with  a  preliminary  exploration  of  the  simplest 
spatio-temporal  masking  paradigm:  The  ability  of  simple  traveling-wave 
sinusoidal  grating  to  mask  another  simple  traveling-wave  grating  of  similar 
orientation.  The  spatial  and  temporal  frequencies  of  the  mask  and  the  test 
gratings  are  the  parameters  of  the  study.  Our  results  (1)  show  that  the 
masking  function  is  symmetrically  band-limited  in  the  relative  velocities 
of  the  two  gratings.  However,  at  any  fixed  velocity  the  masking  function  is 
strongly  asymmetrical  in  spatial  frequency:  masking  is  strongly  sustained 
toward  lower  frequencies  while  at  high  frequencies  masking  declines  rapidly 
and  actually  becomes  replaced  by  facilitation.  These  unexpected  results 
corroborate,  in  a  qualitative  sense,  the  electrophyslological  findings  of 
DeValois  and  Tootell,  on  cat  cortical  "simple"  cells  (2). 

As  part  of  the  more  ambitious  plan  to  study  the  masking  properties 
of  spatio-temporal,  spectrally  3D  band-limited  visual  noise  we  have  addressed 
the  technical  problem  of  generating  and  controlling  such  patterns  in  real¬ 
time.  To  produce  these  dynamic  visual  Images  on  a  C.R.T.  within  the  data 
rate  limitations  of  a  modest  computer  we  are  approximating  continuous  spectra 
by  discrete  (punctate)  components  within  the  desired  3D  spectral  band  of 
spatial  and  temporal  frequency  and  orientation.  The  basic  technical  questions 
are  how  many  such  components  are  required  to  make  a  perceptually  adequate 
approximation  to  the  continuous  3D  spectrum  and  how  should  they  be  distributed. 

These  technical  questions  have  opened  an  unanticipated  and  rich  line 
of  inquiry  into  visual  system  function.  The  issues  are  both  fundamental  and 
subtle.  We  approached  the  study  first  with  combinations  of  discrete. 
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stationary  spectral  components  (i.e.,  the  temporal  frequency,  ffc,  is  zero) 
and  ware  surprised  to  find  that  as  few  as  6  such  components  give  the 
appearance  of  filtered  2D  white  noise  if  the  components  are  properly  chosen 
within  the  pass-band  of  spatial  frequency  and  orientation.  We  have  worked 
out  tentative  rules  for  choosing  these  components  (e.g.  avoiding  repetition 
of  wave  vector  orientation  or  spatial  frequency,  and  avoiding  clustering). 

The  accompanying  figures  show  the  appearances  of  images  composed  of  4,  6,  and 
8  components  to  demonstrate  the  remarkable  efficacy  of  small  nusd»ers  of 
components  to  simulate  a  2D  spatial  stochastic  process.  We  have  adopted  the 
criterion  that  N  components  are  adequately  representative  of  the  continuous 
spectrum  when  an  observer  can  reliably  see  no  difference  between  images  con¬ 
taining  N  and  N+l  components. 

Most  striking  is  the  percept  which  these  band-limited  images  provide: 
that  of  a  "carrier"  wave  component  (representing  the  centroid  of  the  N  components 
in  the  2D  spatial  Fourier  plane,  i.e.,  an  average  spatial  frequency  and 
orientation)  modulated  by  an  overlying  "net"  (representing  changes  of  con¬ 
trast  and  phase).  The  overlying  net  of  2D  amplitude  and  phase  modulation  (2DAPM) 
has  local  dimensions  determined  by  the  difference  frequency  vectors,  i.e., 
the  "beats"  among  the  spectral  components.  If  temporal  frequency  is  added 
to  each  discrete  2D  spatial  frequency  component  the  percept  becomes  even 
more  fascinating,  in  a  way  which  is  impossible  to  appreciate  without  observa¬ 
tion.  Depending  on  how  the  temporal  frequencies  are  assigned,  the  "carrier" 
and  modulation  "net"  may  have  completely  different  apparent  directions  of  motion, 
enhancing  their  dlscriminabillty  and  making  it  easy  to  switch  attention  between 
them.  If  anything,  the  "net",  which  contains  apparent  spatial  frequencies  far 
removed  from  those  actually  present  in  the  image,  is  the  dominant  percept. 

The  fact  that  beat  frequencies  (i.e.,  frequency  difference-vectors)  are  so 
strongly  and  dominantly  perceived  suggests  strong  nonlinearities  in  the  visual 
system.  The  experimental  paradigm  in  which  a  small  number  of  discrete  spectral 
components  are  combined  permits  a  unique  quantitative  approach  to  the  study  of 
these  nonlinear  effects  (4).  We  have  already  been  able  to  show  that  the  very 
simple  beat  pattern  between  two  components  constitutes  a  moderately  effective 


Contract  F49620-K-81-0016 


-3- 


Informal  Scientific  Report 


■ask  of  that  non-existent  spectral  frequency.  We  plan  to  extend  these  studies 
to  tha  more  complex  beat  pattern  of  N  components  (the  2D  "net")  and  in 
particular  we  Intend  to  explore  whether  2D  phase  modulations  are,  by 
themselves,  less  effective  in  masking  than  are  contrast  modulations. 

Nachmlas  and  Rogowitz  (3)  have  carefully  studied  the  very  simple  case  of 
vertical  stationary  gratings  and  find  that  phase  modulations  are  somewhat 
less  effective.  The  modulation  percept  is  so  markedly  different  when  two 
space  dimensions  and  time  are  involved  that  it  would  be  impossible  to 
extrapolate  these  findings  to  it. 

In  summary,  our  original  plan  to  study  the  masking  effects  of  band- 
limited  visual  noise  on  spectral  components  within  or  near  the  3D  '  band-volume 
in  question  has  now  been  expanded  to  consider  effects  on  certain  spectral 
components  far  removed  from  the  band.  These  extended  results  promise  to  be 
very  significant  in  quantifying  the  nonlinear  performance  of  the  human 
visual  system.  Indeed,  the  combination  of  the  traditional  spatio-temporal 
characterization  of  the  visual  system  (generalized  to  three  dimensions) , 
and  our  new  description  in  terms  of  the  response  to  combinations  of  3D 
frequencies,  are  analogs  of  the  first  two  kernels  of  the  full  Wiener 
expansion  of  a  nonlinear  system  (4). 


1.  "Spatlotemporal  Masking:  Asymmetry,  Nonseparability,  and  Facilitation," 

T.  Y.  Zeevi,  R.E.  Kronauer,  and  J.G.  Daugman,  presented  at  the  Annual 
Spring  Meeting,  The  Association  for  Research  in  Vision  and  Ophthalmology, 
May  1982. 

2.  "Spatial  Frequency  Inhibition  in  Cat  Striate  Cortex  Cells,"  K.K.  DeValols 
and  R.B.H.  Tootell,  presented  at  the  Annual  Spring  Meeting,  The  Association 
for  Research  in  Vision  and  Opthalmology,  May  1982. 

3.  "'Masking'  by  Spatially  Modulated'  Gratings,"  J.  Nachmlas  and  B.E.  Rogowitz 
presented  at  the  Annual  Spring  Meeting,  The  Association  for  Research  in 
Vision  and  Ophthalmology,  May  i982. 

Nonlinear  Problems  in  Random  Theory,  N.  Wiener,  MIT  Press,  Cas&ridge, 
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8.3  Appendix  C  -  Technical  Developments 

We  have  developed  the  capability  tc  display  dynamic  two-dimensional 
visual  patterns  which  are  synthesized  in  real-time  from  ensembles  of 
Fourier  components  specified  in  a  3D  spatio-temporal  frequency  space. 

We  can  synthesize  textures  containing  as  many  as  20  such  Fourier 
components,  in  10  co-directional  pairs,  with  independently  specified 
spatial  frequencies,  amplitudes,  phases,  orientations  (with  a  resolution 
of  l/3rd  of  a  degree),  and  temporal  frequencies.  We  can  also  limit  each 
Fourier  component  to  a  spatial  region  defined  by  an  electronic  image 
'window'  having  controllable  2D  location  and  radius  (if  circular)  or 
width  and  length  (if  rectangular). 

The  system  is  fully  interactive  with  a  PDP1134  mini-computer  which 
commands  lists  of  3D  Fourier  components  to  be  synthesized  and  displayed 
on  a  Tektronix  608  monitor.  The  individual  parameters  of  such  textures 
can  be  controlled  manually,  as  well  as  by  computer.  The  images  are 
generated  at  a  frame  rate  of  250  frames/ second . 

We  have  been  developing  a  growing  library  of  computer  programs  which 
accomplish  certain  spectral  operations  in  real-time,  such  as  transforma¬ 
tions  of  a  spectral  cluster  in  the  spatial  Fourier  plane.  These  include 
simple  translation  of  the  spectral  distribution  in  an  arbitrary  direction 
in  the  frequency  domain;  simple  rotation  around  the  cluster's  centroid; 
simple  dilation  relative  to  the  cluster's  centroid;  and  dilation  relative 
to  the  origin  of  coordinates  of  3D  frequency  space.  We  have  ma;’e  &  video¬ 
tape  demonstrating  these  various  spatio-temporal  frequency  transformations, 
using  both  coherent  and  incoherent  spectral  combinations. 

Our  software  developments  allow  us  to  call  up  any  particular  spatio- 
temporal  texture  from  disk  for  controlled  display  to  a  Subject,  who  is  then 
interrogated  in  various  standard  psychophysical  paradigms  such  as  signal 
discrimination  and  detection. 

We  have  integrated  a  second  display  system  into  the  first,  which 
permits  the  simultaneous  display  of  the  spectrum  of  the  image  in  the  Fourier 
plane.  Thus  these  two  displays  can  be  interpreted  as  being  the  2D  Fourier 
transforms  of  each  other,  analogous  to  holographic  representations.  The 
attached  Figure  gives  two  examples  of  such  2D  signals  represented  in  both 
domains.  As  the  spatial  2D  spectrum  of  a  texture  evolves  in  time,  we  see 
simultaneously  the  space-domain  image  and  the  equivalent  spectral 
representation.  Our  dual  display  system  has  the  same  power  and  advantages 
as  seeing  a  voice  spectrogram  evolving  in  time  simultaneously  with  spoken 
speech . 
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